{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wQS9kNoyjsKe"
      },
      "source": [
        "![nebullvm nebuly AI accelerate inference optimize DeepLearning](https://user-images.githubusercontent.com/38586138/201391643-a80407e5-2c28-409c-90c9-327795cd27e8.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Accelerate Fast AI ResNet34 with Speedster"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hBObeC3SmRwl"
      },
      "source": [
        "Hi and welcome 👋\n",
        "\n",
        "In this notebook we will discover how in just a few steps you can speed up the response time of deep learning model inference using the open-source library nebullvm."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%env CUDA_VISIBLE_DEVICES=0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87jOeOOtktQy"
      },
      "source": [
        "### Fine-tune a fast.ai model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XlVUVGOAlS6O"
      },
      "source": [
        "For the tutorial, we will use a fast.ai notebook for beginners in which we will classify whether the input image contains a cat (True label) or a dog (False label). Let's jump to the code.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9cFt-FEvlNkG"
      },
      "outputs": [],
      "source": [
        "from fastai.vision.all import *"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GqdMEBPZlmpu",
        "outputId": "18d8a166-9b5d-4c91-cbc7-c8591bd5c0d2"
      },
      "outputs": [],
      "source": [
        "path = untar_data(URLs.PETS)\n",
        "files = get_image_files(path/\"images\")\n",
        "\n",
        "def label_func(f): return f[0].isupper()\n",
        "\n",
        "dls = ImageDataLoaders.from_name_func(path, files, label_func, item_tfms=Resize(224), num_workers=0)\n",
        "dls.show_batch()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VrmI4VeZlhJG"
      },
      "source": [
        "After downloading a sample of images of dogs and cats, we fine-tune the fast.ai model.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MJ8q9xxBlv1x",
        "outputId": "8169f902-3dd0-449c-c293-91fb7ab94003"
      },
      "outputs": [],
      "source": [
        "learn = cnn_learner(dls, resnet34, metrics=error_rate)\n",
        "learn.fine_tune(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RBzr8_47lxsW",
        "outputId": "b87781d6-2826-4cc6-9fd3-57da5cdcbbd4"
      },
      "outputs": [],
      "source": [
        "valid_loss, error = learn.validate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WSWq0il6l0eC"
      },
      "source": [
        "Now that we have fine-tuned the model, let's calculate the time required to run a prediction as an average over 100 tests.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o_iMOqI_l6-Y"
      },
      "outputs": [],
      "source": [
        "import time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JNZXAgIYl883"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "times = []\n",
        "for _ in range(100):\n",
        "    st = time.time()\n",
        "    preds = learn.predict(files[0])\n",
        "    times.append((time.time()-st)*1000)\n",
        "fastai_vanilla_time = sum(times)/len(times)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N9IDkfyDmADn",
        "outputId": "0113620d-4c77-4a9f-ae1e-e64b0cb32293"
      },
      "outputs": [],
      "source": [
        "print(f\"Average prediction time: {fastai_vanilla_time} ms,\\nPrediction: {preds}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hlwl87jRmBy2"
      },
      "outputs": [],
      "source": [
        "#learn.save(\".\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bes-NoZnmhyy"
      },
      "source": [
        "### Install nebullvm"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "48aljCHu14-H",
      "metadata": {
        "id": "48aljCHu14-H"
      },
      "source": [
        "Install nebullvm:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "QFQh3BVr1-GO",
      "metadata": {
        "id": "QFQh3BVr1-GO"
      },
      "outputs": [],
      "source": [
        "!pip install speedster"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8a7a86b3",
      "metadata": {
        "id": "8a7a86b3"
      },
      "source": [
        "Install deep learning compilers:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cffbfa32",
      "metadata": {
        "id": "cffbfa32"
      },
      "outputs": [],
      "source": [
        "!python -m nebullvm.installers.auto_installer --frameworks torch --compilers all"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Data preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zVfy0VBooG_J"
      },
      "source": [
        "Now we prepare the dataset so that it can be processed by Speedster."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RuUavpyooIBT"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "\n",
        "xs, ys = [], []\n",
        "for i, (x, y) in enumerate(dls.train):\n",
        "    if i >=100:\n",
        "        break\n",
        "    xs.append(x)\n",
        "    ys.append(y)\n",
        "xs = torch.cat(xs, dim=0)\n",
        "ys = torch.cat(ys, dim=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kkVzQVmgoMQh"
      },
      "outputs": [],
      "source": [
        "dl_nebullvm = [((x.unsqueeze(dim=0),), y.unsqueeze(0)) for x, y in zip(xs, ys)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_Eb_AAeqoOUS"
      },
      "outputs": [],
      "source": [
        "original_model = learn.model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0siBvWcsnv49"
      },
      "source": [
        "### Unconstrained without accuracy loss (thus constrained)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ToxCH47qstn9"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchvision.models as models\n",
        "from speedster import optimize_model, save_model, load_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "njoWqCSzvzpr"
      },
      "outputs": [],
      "source": [
        "# Load a resnet as example\n",
        "model = original_model\n",
        "\n",
        "# Provide an input data for the model    \n",
        "input_data = dl_nebullvm\n",
        "\n",
        "# Run Speedster optimization\n",
        "optimized_model = optimize_model(\n",
        "  model, input_data=input_data, optimization_time=\"unconstrained\",\n",
        ")\n",
        "\n",
        "# Try the optimized model\n",
        "# x = torch.randn(1, 3, 224, 224)\n",
        "# res = optimized_model(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GGRbJL6Xq6Ns"
      },
      "outputs": [],
      "source": [
        "optimized_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h75V23FSs2MZ"
      },
      "outputs": [],
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Set the model to eval mode and move it to the available device\n",
        "model.eval()\n",
        "model.to(device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R_QrrT0oq1i_"
      },
      "outputs": [],
      "source": [
        "res_optimized = optimized_model(x)\n",
        "res_optimized"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xtjV8pDYxIIl"
      },
      "outputs": [],
      "source": [
        "from nebullvm.tools.benchmark import benchmark\n",
        "\n",
        "benchmark(model, input_data)\n",
        "benchmark(optimized_model, input_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWJCMGGJxaG5"
      },
      "source": [
        "### Unconstrained with 2% accuracy loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g9Huil4-xeX5"
      },
      "outputs": [],
      "source": [
        "# Load a resnet as example\n",
        "model = original_model\n",
        "\n",
        "# Provide an input data for the model    \n",
        "input_data = dl_nebullvm\n",
        "\n",
        "# Run Speedster optimization\n",
        "optimized_model = optimize_model(\n",
        "  model, input_data=input_data, optimization_time=\"unconstrained\", metric_drop_ths=0.02, metric=\"accuracy\"\n",
        ")\n",
        "\n",
        "# Try the optimized model\n",
        "# x = torch.randn(1, 3, 224, 224)\n",
        "# res = optimized_model(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cLxoOzxe4clI"
      },
      "outputs": [],
      "source": [
        "# Set the model to eval mode and move it to the available device\n",
        "model.eval()\n",
        "model.to(device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c3QvxwUD4clI"
      },
      "outputs": [],
      "source": [
        "optimized_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dRLd4QMJ4clI"
      },
      "outputs": [],
      "source": [
        "benchmark(model, input_data)\n",
        "benchmark(optimized_model, input_data)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "ceb60d8c",
      "metadata": {
        "id": "ceb60d8c"
      },
      "source": [
        "## Save and reload the optimized model"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "d9eda1a0",
      "metadata": {},
      "source": [
        "We can easily save to disk the optimized model with the following line:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "62b6fcbf",
      "metadata": {},
      "outputs": [],
      "source": [
        "save_model(optimized_model, \"model_save_path\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "3c968d51",
      "metadata": {},
      "source": [
        "We can then load again the model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c1340c49",
      "metadata": {},
      "outputs": [],
      "source": [
        "optimized_model = load_model(\"model_save_path\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "b77ff2ac",
      "metadata": {
        "id": "b77ff2ac"
      },
      "source": [
        "<center> \n",
        "    <a href=\"https://discord.com/invite/RbeQMu886J\" target=\"_blank\" style=\"text-decoration: none;\"> Join the community </a> |\n",
        "    <a href=\"https://nebuly.gitbook.io/nebuly/welcome/questions-and-contributions\" target=\"_blank\" style=\"text-decoration: none;\"> Contribute to the library </a>\n",
        "</center>\n",
        "\n",
        "<center> \n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#key-concepts\" target=\"_blank\" style=\"text-decoration: none;\"> How speedster works </a> •\n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#documentation\" target=\"_blank\" style=\"text-decoration: none;\"> Documentation </a> •\n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#quick-start\" target=\"_blank\" style=\"text-decoration: none;\"> Quick start </a> \n",
        "</center>"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.8.10 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.9 (default, Apr 13 2022, 08:48:06) \n[Clang 13.1.6 (clang-1316.0.21.2.5)]"
    },
    "vscode": {
      "interpreter": {
        "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
